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ABSTRACT 

To explore the issue of performing a non-interactive numerical weather forecast with 
an operational global model in assist of astronomical observing, we use the Xu-Randall 
cloud scheme and the Trinquet- Vernin AXP seeing model with the global numerical 
output from the Global Forecast System to generate 3-72h forecasts for cloud coverage 
and atmospheric seeing, and compare them with sequence observations from 9 sites 
from different regions of the world with different climatic background in the period of 
January 2008 to December 2009. The evaluation shows that the proportion of prefect 
forecast of cloud cover forecast varies from ~ 50% to ~ 85%. The probability of cloud 
detection is estimated to be around ~ 30% to ~ 90%, while the false alarm rate is 
generally moderate and is much lower than the probability of detection in most cases. 
The seeing forecast has a moderate mean difference (absolute mean difference < 0.3" in 
most cases) and root-mean-square-error or RMSE (0.2"-0.4" in most cases) comparing 
with the observation. The probability of forecast with < 30% error varies between 40% 
to 50% for entire atmosphere forecast and 30% to 50% for free atmosphere forecast for 
almost all sites, which being placed in the better cluster among major seeing models. 
However, the forecast errors are quite large for a few particular sites. Further analysis 
suggests that the error might primarily be caused by the poor capability of GFS/AXP 
model to simulate the effect of turbulence near ground and on sub-kilometer scale. In 
all, although the quality of the GFS model forecast may not be comparable with the 
human-participated forecast at this moment, our study has illustrated its suitability for 
basic observing reference, and has proposed its potential to gain better performance 
with additional efforts on model refinement. 

Subject headings: Astronomical Phenomena and Seeing: general — Astronomical Phe- 
nomena and Seeing: individual(astronomical observing, numerical meteorological fore- 
cast, cloud cover, astronomical seeing) 
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Introduction 



Almost all kinds of ground-based astronomical observations, especially for optical ones, are 
extremely dependent on meteorological condition, so it is no doubt that making the meteorological 
prediction as precise as possible would significantly help observers to schedule the observation and 
improve the efficiency of telescope operation. Among all meteorological variables, cloud amount or 
cloud cover is apparently the do minant fact or, while atmospheric/astronomical seeing, or sometimes 
described as Fried parameter (|Friedlll965l ). is also important. 



A climatologically study to the annual values of cloud cover and/or atmospheric seeing for a 
proposed professional observatory is usually d one in the "site survey" prior to the construction (see 



Walker 



1970 



Fuentes &: Muhoz-Tuhonlll990l . for examples). However, to maximize the observing 



resources, astronomers not only need to know the approximate percentage of clear nights in a year, 
but also wish to know whether the sky will be clear or not in the next a few nights. In another 
word, they expect weather forecast to be as accurate and precise as possible. However, since 
astronomical observatories are generally built in distant areas, therefore with sparse meteorological 
observations available and less interests from meteorologists, special forecast that aim at assisting 
astronomical observation had not been widely practiced until very recent years. Since the end of 
1990s, special forecast services and products based on meso-scale regional numeric models and/or 
real-time satellite images have been d eveloped at the large professional observatories, such as Mauna 
Kea Weather Center or MKWC (see iBusinger et al.ll2002l. for an over view) and the nowcast model 
at European Southern Observatory (ESO) ([Erasmus fc Sarazinll200ll ). 



As the operation of high resolution numeric models would require the ability to perform speedy 
computation (which is usually only available at large professional observatories), attempts aiming 
at making direct uses of the model "fields" from global/continental models were carried out later, 
such as the Clear Sky Charfl that uses Canadian Meteorological Centre's Global Element Multi- 
scale (GEM)@ since 2002 (iDankd 120031 ) . and our 7Timer syste rr| that uses National Centers for 



Envi r onmental Prediction (NCEP)'s Global Forecast Syste nfl (refer as GFS hereafter; see ISelal 



1980l :l Whitaker et al.l 120081 . for an overview of the model) since 2005. These "direct-from-model" 
forecasts only have decent spatial and vertical resolution (for a comparison, the spatial resolution 
of the GFS model is about 40km, while the regional model operating at MKWC can reach 1km), 
but it doesn't require heavy computation works either: the retrieval of the model fields can be 
done with an Internet-connected Personal Computer (PC) within a couple of minutes, making it 
the most favorable and probably the only choice when speedy computer is not available and the 
demand on forecast precision/accuracy is not critical. 



: http : //cleardarksky . com/ csk/ 

z http : //collaboration. cmc .ec.gc.ca/ science/rpn/gef _html_public 
:i http : 777timer . y234 . cn| 

^http: //www . emc . ncep .noaa. gov/modelinf o/ 
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Interestingly, although these services have been put into good use by private, public and 
even some professional observatories, there is no quantitative and systematic understanding on 
how accurate the model fields are up to now. For example, t he only reported es t imati on of the 
accuracy of cloud field forecast for a global model was done by Erasmus fc Sarazinl (|200ll ) in 1992- 
1993, which suggested that only 15-25% of cloudy nights could be identified with the European 
Centre for Medium-rage Weather Forecasting model (ECMWFfl. This is age old considering there 
had been a number of significant upgrades of the global models in the following decade. The 
forecast for atmospheric seeing, on the other hand, is more complex, since it is related to vertical 
"fine structure" of the atmospheric column and is not directly provided as part of output in any 
global model. In general, there are two tracks for atmospheric seeing forecast: "nowcast" track 
using near real-time meteorological observation profile combining with a statistical model (such as 



Mur tagh et al 



( Tatarski 



1961 



1995 ); or the "model" track either using the derivations of the Tatarski's f ormul a 



Coulman et al .11 198a ) or the numeric model proposed by ICoulman et al.l (jl986l ). 



The first track is relatively intuitive and is accurate enough in many occasions, but it has a very 
short forecast range (usually less than 24h) and heavily depends on the availability and quality 
of the observational data; for the second track, one would need to divide the atmospheric column 
into a good number of layers to gain a numeric simulation close enough to the actual situation, 
which will again require assistance from a speedy computer. In order to solve these shortcomings, 
Trinquet Verninl ((2006) takes advantages from both tracks and introduced the AXP model. With 
that model, one only needs to divide the atmospheric column by a number close to that available 
in most global models, and the consistency from the simulation of the AXP model to the actual 
situation is satisfying according to the authors. In all, these "direct-from-model" forecasts can be 
a practical solution for the observers without the ability to operate a high precision regional model 
by their own, and the job to do is to assess how accurate these forecasts are. 

We organize this paper as follow. In Section 2, we briefly outline the technical details of the 
GFS model that used in this study as well as the Xu-Randall cloud scheme that it used for cloud 
simulation and the AXP model that we used to derive forecast of atmospheric seeing. In Section 
3, we describe the observations we used to evaluate the GFS model. Section 4 presents the details 
and discussions of evaluation methodology and result; while Section 5 gives the concluding remarks 
of this study. 



2. Forecast 

2.1. The NCEP GFS model 

The GFS model provides output in two grids with different spatial resolution: grid 003 at 
1° x 1°, and grid 004 at 0.5° x 0.5°. To provide best-possible forecast, we use the later in our study. 



: //www . ecmwf . int/ 
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Model outputs from the period of January 1, 2008 to December 31, 2009 at three-hourly interval 
for < r < 72h at 00Z initialization is retrieved from the National Operational Model Archive & 
Distribution System (NOMADS; see lRutledge et al.ll2006l ) for evaluation. 



The GFS dataset contains approximately 140 fields, supplying forecast fields both of general 
meteorological interests (such as temperature, humidity, wind direction and speed, etc.) and for 
special purpose, including cloud cover fraction on different layer (low, mid, high, convective, and 
of total atmospheric column). Although the atmospheric seeing is not among the output fields, it 
can be derived indirectly as every of the required meteorological variables are given. 



2.2. Cloud scheme 



In the G FS model, cloud cove r fraction for each grid box is computed using the cloud scheme 
presented bv lXu &: Randalll (|1996l ). which is shown as eq. [1]. In this equation, RH is the relative 
humidity, q* and q c are the saturation specific humidity and q cm in is a prescribed minimum threshold 
value of q c . Depending on the environmental temperature, q* and q c are calculated with respect to 
water phase or ice phase (Yang, personal communication). Cloud cover fraction can therefore be 
calculated for any layer as long as the RH, q* and q c are known and qcmin 

is suitably prescribed. 

We note that the calculation is done as part of the model simulation at NCEP, so the cloud fields 
are used as-is from the GFS datasets. 



2000(q e 



C = max[i?#4(l - e 



T 

mm{max[((l-flfl")q*) 1 ,0.0001], 1.0} 



),0.0] 



(1) 



The GFS model divides the whole atmospheric column into 26 layers. The total cloud cover for 
entire atmospheric col umn is derived un der the assumption that clouds in all layers are maximally 
randomly overlapped ([Yang et al.ll2005l ). 



2.3. Seeing model 



The way atmospheric optical turbulence affects as tronomical obs e rving is theor e tically de- 



1961 



Roddier 



1981 



Tokovinin 



scrib ed by the Kolmogorov-Tatarski turbulence model (ITatarski 
2002) which suggested only one parameter is needed to describe the quality of atmospheric seeing 
€q with A to be the wavelength associated with seeing (A = 5 x 10~ 7 m in most cases) and rg to be 
the Fried parameter:: 



e = 0.98— 



(2) 



The Fried parameter, tq, is defined as followed in the direction of zenith (|Coulmanlll985l ). with 
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Zq to be the geopotential height of the observing site, to be the refractive index structure co- 
efficient indicating the strength of turbulence associated with the temperature structure coefficient 
Cj., P to be the pressure in hPa and T to be the temperature in Kelvin: 



Itt f'°° % 

r = [0.423(— f / C 2 N dZ]s (3) 
A JZo 

cS ._ cf( Z£2^f ) . (4) 

Therefore, with the above formulas and suitable inputs, we can derive the total effect of 
atmospheric turbulence from the integral of C^(Z) for all atmospheric layers. 



CJ = r(x) - T f X + r) (5) 
|r| 3 

There are several ways to derive Cj.. A theoretical approach is shown as eq. [5], where x and 
r to be the position and separation vector, respectively. However, by theory |r| should be around a 
few 0.1m to precisely describe the effect of turbulence (which is not practically possible to date for 
model simulation as it would require one to use about 100,000 layers for model simulation). The 
AXP model using an alternative approach by considering a simple expression of C^(h) as follow, 
with the power p{h) adjusts the amplitude of peaks and A(h) connects the level: 



C| = (C*)(h)[A(h)™ z yW (6) 

The potential temperature 9 in eq. [6] can be calculated by Poisson's equation with T to be 
the absolute temperature of a parcel in Kelvin and P to be the pressure of that air parcel in hPa: 



= T( ™)0.286 (7) 

The authors of the AXP model then statistically determine the values of the three coefficients, 
(Cj,)(h), A(h), and p(h), by a vertical spacing of 1km up to an altitude of 30km, based on airborne 
observations from 162 flights at 9 sites during 1990-2002. On the other hand, the coordinate 
system adopted by the GFS model is P-coordinate system, which divides the atmospheric column 
by pressure. The vertical spacing of the GFS model over low and mid level atmosphere is around 
lkm and is roughly compatible with the Z-coordinate system adopted by the AXP model, however 
the former becomes too sparse at high level atmosphere (as illustrated in Table [T]) . To solve this 
problem, we set up a "degeneracy" scheme to allow using the AXP model coefficients in P-coordinate 
system by weighting the values according to the correlation between atmospheric pressure and 
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altitude denned by U.S. Standard Atmosphere in 19 7 fl The transformed eoeffieients for each 
pressure layer are given in Table [2j As the output of GFS model provides fields of temperature and 
atmospheric pressure for each of its vertical layer, we can align the fields with eq. [2], eq. [3], eq. 
[4], eq. [6] and eq. [7] to derive final eo in form of arcsec. We refer this hybrid model as GFS/AXP 
model hereafter. 

Trinquet &: Vernin reported an accuracy of 58% of the original AXP model forecasts with error 
within ±30% from observations. However, as we have modified the model layers to fit the GFS 
model, some additional errors may have been induced. One may expect two possible sources of 
errors caused by the degeneracy. The first kind of possible error comes from the layers higher than 
10km. As the GFS model layers with altitude > 10km have a vertical thickness significantly larger 
than 1km, one may suggest that some performance may be lost due to data roughness. However, we 
argue that the loss of performance by this reason should be minimal, as the high level atmosphere is 
much less active than the mid or low level atmosphere and contribute minimal turbulence to seeing 
than the later twc@, so a vertical spacing degeneration from 1km to 2km at high level atmosphere is 
unlikely to induce error of significance. On the other hand, the degeneracy in planetary boundary 
layer or PBL, which serves as the second possible source of error, may induce something large. PBL 
is a layer that directly influenced by the atmosphere-ground interaction, and it has been shown 



that the PBL turbulence contributes a major part to the seeing during the night (jAbahamid et al 



2004). To improve the model performance at PBL, the AXP model divides the planetary boundary 
layer (PBL) into three sub-layers: 0-50m above ground, 50-100m above ground, and 100-1, 000m 
above ground. However, the GFS model only provides three near-ground fields, which is at 2m 
above ground, 0.995<r (~ 100m above ground) and 30hPa above ground (~ 400m), we have to use 
identical dO/dz computed from 2m above ground and 0.995c for layers of 0-50m and 50-100m and 
the d9/dz computed from 0.995a and 30mb above ground for layer of 100-1, 000m, the ignorance 
of data at 50m would almost certainly to induce some error. To work around this issue and allow 
a direct assess of the GFS/AXP model, we also generate seeing forecast for free atmosphere only - 
i.e. with PBL (< 100m in our study) excluded^! - for our evaluation. The free atmosphere forecast 
will be compared with the observations from Multi- Aperture Scintil lation Sensor (MASS) which is 



designed to measure the seeing in free atmosphere (jTokovininl 120021 ) 



6 See |http : //ntrs . nasa . gov/archive/nasa/casi . ntrs . nasa . gov/1 9770009539_1977009539 . pdf 

7 Study bv lLi et alj (|2003h suggests the atmosphere beyond lOOhPa (~ 15,000m) contributes less than 0.01" to the 



seeing. 

8 We do not exclude the 100m-l,000m region in our free atmosphere seeing forecast because the MASS instrument 
using for measurement of "free atmosphere" seeing considers an altitude of 500m above ground as PBL top limit, 
and includes measurements of layers at an altitude as low as 500m and 1,000m above ground. 
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3. Observation 



We collect cloud cover and seeing observations for a total of nine sites from Central/East 
Asia, Hawaii and Central/South America in the period of January 2008 to December 2009. The 
respective information of each site including the PBL top limit and P(h) used in the GFS/AXP 
model are listed in Table El 

As these observations are all made in sequence with sampling frequency around l-1.5min except 
Nanshan and Lulin (which will be deal separately and will be described below), they are firstly 
processed to match the time interval of the GFS model output (which is 3h). To ensure that the 
observation is representative in the corresponding interval, we set a minimum data points of 100 
for each interval. A sampling frequency of l-1.5min corresponds 120-180 data points per 3h, so the 
minimum threshold of 100 is reasonable. 

As all seeing observations are obtained either by Differential Image Motion Monitor (DIMM) 
or MASS, they can be used without further reduction since they give the measurement of eo directly. 
On the other hand, the cloud cover observations from Paranal, Nanshan and Lulin are obtained 
with different tools, so they must be reduced to the same definition with the GFS model output 
before comparing them with the later. The reduction procedure is described below. 



Paranal The cloud sensor installed at Paranal determines the sky condition by measuring the 
flux variation of a star. The sensor graph will suggests a possible "cloudy" condition when the 
root-mean-square (RMS) of the flux variation is larger than O.ofl 



Nanshan For Nanshan, the operation log is used to verify the sky condition. The operation 
log includes the image sequence log and observer's notes. We divide each night into two "parts": 
evening and morning. A "part" with roughly > 50% observable time (with images taken and 
indication of good observing condition from the observer) would be marked as "clear", otherwise 
it would be marked as "cloudy". 



Lulin A Diffraction Limited Boltwood Cloud Sensor is installed at Lulin to produce sequence 
observations. The cloud sensor determines the sky condition by comparing the temperature of the 
sky to ambient ground level temperature, and a difference threshold of — 25°C is set to distinguish 
"clear" and "cloudy" conditions 10 !. 



9 See http : //archive . eso . org/asm/ambient-server 

10 See http : //www. cyanogen. com/downloads/f iles/claritymanual .pdf 
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4. Evaluation 

4.1. Evaluation of cloud cover forecast 

The preliminary result from our companion study suggests that roughly 70% cloud cover 
forecasts from the GFS model can achieve an error less than 30% (Ye & Chen, in prep.) at 
0h< r < 72h at grid with spacing of ~ 300km. However, the cloud cover observations used in 
this study are all made at single geographic points and are categorical (either clear or cloudy) 
rather than over a grid area and being quantitative. So in order to make comparison between the 
forecasts and the observations, we first need to simplify the forecast into two categories by setting 
a spatial threshold that divides "cloudy" and "clear" situations. To examine the result sensitivity 
with different thresholds, we set the threshold at 30%, 50% and 80%, respectively. Considering 
most astronomical observatories are placed at areas with higher chance to be clear rather than 
cloudy, we should focus the "cloudy" event in our study rather than the "clear" event. Thus, event 
"cloudy" is set to be the "yes" statement, while "clear" is set to be the "no" statement. 

We use four indicators to evaluate the performance of the forecast^: Proportion of Perfect 
Forecasts (PPF), Probability of Detection (POD), False Alarm Rate (FAR), and Frequency Bias 
Index (FBI). Let H donates "hits" (forecasted and observed), F donates "false alarms" (forecasted 
but not observed), M donates "missed" (not forecasted but observed), and Z donates to not- 
forecasted and not-observed situation, we have: 

PPF H + Z 



POD -- 
FAR 
FBI = 



H + F + M + Z 
H 



H + M 
F 

' F + Z 
H + F 
H + M 



The H, F, M and Z numbers for each of the three sites are listed in Tabled and the comparison 
results between the forecast and observation are shown in Table [5) 

As the result from our companion study has implied that the accuracy declination with the 
increase of r is small (Root-mean-square-error or RMSE varies within 5% for r up to 72h) for the 
GFS model, we see no need to list percentages for each night separately. The percentages given 
in Table [5] are the average of 0h< r < 72h, with the uncertainty ranges donate by the nights 
with highest/lowest percentages. The large uncertainty of the FBI of Paranal is caused by small 
dominators since the numbers of "false alarms" are 5-10 times more than the numbers of "misses". 



1 A more detail technical document about the application of these four indicators in meteorology may be found at 
http: //www. ecmwf . int/products/f orecasts/guide/Hit_rate_and_False_alarm_rate .html 
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Although PPF is a biased score since it is strongly influenced by the more common category, 
it gives a rough indication about the typical percentage of "correct" forecast for one site. A clear 
feature revealed by Table [5] is the climate-dependence of the PPF: arid sites tend to have higher 
PPF, while humid sites tend to have lower. Generally speaking, the PPF varies from 50% to 85% 
for the three sites in our sample. As the sites have covered both the extremes of climate type (from 
arid and humid), it can be considered that this result is relatively representive. 

The result revealed by POD is also encouraging. Even for Paranal, the site with an annual 
cloudy percentage as low as 10% (jArdeberg et al.lll990i ). the POD can still varies betw een 20-60% 



and being better than the 15-25% percentage reported by ([Erasmus &: Sarazinl l200ll ). We also 



note that the forecast FAR for Paranal and Nanshan is less than half of the forecast POD, far 
from as good as the human-participated forecasts (for reference, the MKWC forecaster which can 
reach a FAR as low as 193 h. but is still reasonable for basic observing reference. We note that 
the forecast FAR for Lulin is very large; however, considering that Lulin is surrounded by a very 
complex terrain, where the elevation variation in its belonged grid box is as large as 3,500m, it is 
not unexpected for a badly-looking FAR. In final, the FBI for all sites are larger than 1, suggesting 
the GFS model tends to make more "false alarms" rather than "misses". 



4.2. Evaluation of seeing forecast 

The evaluation of seeing forecast is relatively easier than that of cloud cover forecast since 
the forecasts and observations are in same definition, and we can simply compare the mean and 
root-mean-square-error (RMSE) of the difference between the forecasts and observations (Tabled]). 
The mean errors are computed by averaging the differences between instantaneous forecasts and 
observations. As the increase of error is small (ARMSE<~ 0.05") for r up to 72h, we combine 
forecasts and observations from all the three nights into a whole for our study. 

As seen in Table [6l either entire atmosphere forecast or free atmosphere forecast has moderate 
error comparing with the observation, while the former tends to slightly overestimate the value 
and the latter tends to the opposition. However, the tendency is severe for the cases such as Cerro 
Pachon which the mean difference reads +0.46". A possible explanation is that many observatories 
are located in mountainous areas and being actually much higher than the height of its belonged 
grid used in GFS models (the height "differences" vary from 0-4, 000m among our sample, as shown 
in Table [3]); this may produce error in PBL seeing estimation. However, we find no substantial 
support for this assumption as there is no significantly tendency between the forecast bias and the 
height difference (Figure [1]). Dramatically, the site with smallest height difference (Cerro Pachon, 
with a height difference of 9m) has the largest mean error (+0.46"). 

Another equally plausible explanation is the combined effect of the poor consistency between 



See http://mkwc.ifa.hawaii.edu/forecast/mko/stats/index.cgi?night=l Mcster=fcsts&var=fog&cut=2. 
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the GFS/AXP model versus the actual Cff variance in PBL and error induced by the layer degen- 
eration of the AXP model for high altitude areas. In fact, we do find a weak bias tendency for free 
atmosphere forecast as shown in Figure [2j the forecast for low-altitude sites tends to be better and 
all sites below 3,000m have an absolute mean forecast error below 0.25". This feature fits the fact 
that the airborne data used to determine the coefficients in the AXP model only includes sites with 
altitude up to 2,835m. But generally speaking, no substantial correlation between bias tendency 
and an unique geographic modeling factor can be identified, therefore we can only suggest that the 
bias is contributed by a combination effect of some factors. 

To give a more comprehensive understanding on the quality of the GFS/AXP forecast, we plot 
the cumulative distribution of the relative forecast error for each site as in Figure [3] (forecast for 
entire atmosphere) and Figure H] (forecast for free atmosphere only). Let eb to be the forecasted 
seeing value and eo to be the observed seeing value, the relative forecast error E(eo) is calculated 
by 

E(eo) = 1 60 " 60 1 



From the two figures, we can identify that the probabilities of producing forecast with < 30% 
error concentrate in 40-50% for entire atmosphere seeing forecast and 30-50% for free atmosphere 
seeing forecast. By contrast, Trinquet & Vernin gives a probability of 58% and 50% for the original 
AXP model to produce entire atmosphere or free atmosphere forecast in the same quality. Generally 
speaking, our result is in rough agreement with them, although the model's performance is rather 
unsatisfying on a few particular sites. 

In addition to the summarizing table and cumulative distribution figures, we also present the 
forecast-observation distributions for each site as in Figure [5] and Figure El We can see that although 
the statistical values might be satisfying, the forecast-observation distribution figures imply that the 
co rrelation between forec ast and observation is poor. This is not out-of-expected, since the study 



of ICherubini et al.1 (120091 ) at MKWC has suggested that one may achieve a good approximation 
of the actual condition with ~ 80 vertical layers up to lOhPa, the ~ 15 layers used in our model 
may be far from sufficient for a well-correlated distribution. Our assumption is reinforce by the 
fact that forecasts for free atmosphere seeing are generally concentrated in a narrow region despite 
their mean is close to that of observation, as shown in Figure [6) This phenomenon implies that 
turbulence over sub-kilometer scale in the vertical direction might be the major contributor to a 
bad seeing condition in free atmosphere region. 

In final, we compare our result with several major models/forecasters (Table [7J). The RMSE 
and < 30% error probabilities for the original AXP model, Vernin- Tatarski model, /seeing 
median and seeing mean are derived by Trinquet & Vernin with their experimental profile obser- 
vations. The MKWC/WRF model is initiated with the GFS model output, but the simulation 
would eventually derives output with final grid spacing of 1km and a vertical layer number of 40, 
resulting a forecast RMSE at 0.36" for the 1st night. As revealed by TableO the RMSE uncertainty 
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ranges of GFS/AXP model just fall between the mean of MKWC forecaster and the original AXP 
model, while the < 30% error probability is in the better cluster in all models under most cases. 
Interestingly, a "direct" comparison of the GFS/AXP model with the MKWC forecaster over the 
seeing forecast for the same site (Mauna Kea) even slightly favors the former (0.26" versus 0.28" 
for three-night mean). In short, this result has highlighted the potential of the GFS/AXP model 
to be a competitive forecast tool once the layer degeneracy and high-altitude issues are solved. 

5. Conclusion 

We have carried out a comprehensive study over the topic of performing automatic numeric 
forecast of cloud cover and atmospheric seeing with the Global Forecast System, an operational 
global model. Sequence observations on cloud cover and atmospheric seeing from 9 sites from 
different regions of the world with different climatic background in the period of January 2008 to 
December 2009 are used to evaluate the forecast. Although the performance of the model forecast 
may not be comparable with the human-participated forecast, our study has shown that the forecast 
to be acceptable for basic observing reference. Our study has also reveal the possibility to gain 
better performance from the model with additional efforts on model refinement. 

For cloud cover forecast, we have found that the proportion of perfect forecast varies from 
~ 50% to ~ 85% for all three sites we evaluated, including a site located in subtropical region with 
a very humid climate (Lulin). In particular, we have found that the model is capable to detect a 
significant amount of occurring clouds while the false alarm rate is moderate. The probability of 
detection is still measured to be 20-60% even for site with very low cloudy probability (Paranal). 

For atmospheric seeing forecast, we adopted the AXP model introduced by Trinquet & Vernin. 
We found that forecast for entire atmosphere tends to slightly overestimate the seeing, while the 
free atmosphere forecast tends to the opposition. The RMSE for free atmosphere seeing is smaller 
(0.22"-0.42") than that of entire atmosphere (0.26"-0.50"), but both values can indicate a decent 
quality of the forecast comparing with the other major models. Further analysis suggests that a 
major contributor to the forecast error might be the layer degeneracy issue of the hybrid GFS/AXP 
model. On the other hand, the probability of GFS/AXP forecasts with < 30% error varies between 
40-50% for entire atmosphere forecast and 30-50% for free atmosphere forecast in most cases, which 
is in the better cluster among major seeing models. To conclude, our study has suggested a decent 
performance of the GFS/AXP model that is suitable for basic observing reference. Our study 
has also suggested that the model has the potential to become a rather useful forecast tool with 
additional efforts on model refinement. 
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Fig. 1. — Absolute forecast mean errors and the height differences between actual height and GFS 
grid for all sites for entire atmosphere (upper) and free atmosphere (lower). Abbreviations are: P 
- Paranal, SPM - San Petro Martir, MK - Mauna Kea, CO - Cerro Tololo, CA - Cerro Armazones, 
CT - Cerro Tolonchar, and CP - Cerro Pachon. 
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Fig. 2. — Absolute forecast mean error and the altitude of the site for all sites with DIMM obser- 
vation (upper) and MASS observation (lower). Abbreviations are: P - Paranal, SPM - San Petro 
Martir, MK - Mauna Kea, CO - Cerro Tololo, CA - Cerro Armazones, CT - Cerro Tolonchar, and 
CP - Cerro Pachon. 
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Fig. 3. — Cumulative distribution of relative error of seeing forecast for ground layer plus free 
atmosphere. Dashed line represents a relative error of 30%. The probability of < 30% forecast error 
is 63% for Paranal, 47% for Mauna Kea, 45% for San Pedro Martir, 40% for Cerro Armazones, 
14% for Cerro Pachon, and 29% for Cerro Tolonchar, respectively. The Mauna Kea curve is quite 
abnormal due to small sample number (n = 34) and we estimate that the < 30% forecast error 
probability is about 15% too low. 



-17- 




Fig. 4. — Cumulative distribution of relative error of seeing forecast for free atmosphere only. 
Dashed line represents a relative error of 30%. The probability of < 30% forecast error is 21% for 
Mauna Kea, 47% for San Pedro Martir, 30% for Cerro Tololo, 38% for Cerro Armazones, 40% for 
Cerro Pachon, and 34% for Cerro Tolonchar, respectively. The Mauna Kea curve is quite abnormal 
due to small sample number (n = 42) and we estimate that the < 30% forecast error probability is 
about 15% too low. 
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Table 1. Layers of the GFS model (in P-coordinate system) and corresponded altitude layers (in 

Z-coordinate system). 



GFS layer Corresponding altitude 



2m above ground 
0.995<7 a 
30hPa above ground 
900-850hPa 
850-800hPa 
800-750hPa 
750-700hPa 
700-650hPa 
650-600hPa 
600-550hPa 
550-500hPa 
500-450hPa 
450-400hPa 
400-350hPa 
350-300hPa 
300-250hPa 
250-200hPa 
200-150hPa 
150-100hPa 

100-70hPa 

70-50hPa 

50-30hPa 

30-20hPa 

20-10hPa 



2m above ground 
~ 100m above ground 
~ 400m above ground 
988-l,457m 
l,457-l,948m 
l,948-2,465m 
2,465-3,011m 
3,011-3,589m 
3,589-4,205m 
4,205-4,863m 
4,863-5,572m 
5,572-6,341m 
6,341-7,182m 
7,182-8, 114m 
8,114-9,160m 
9,160-10,359m 
10,359-ll,770m 
ll,770-13,503m 
13,503-15,791m 
15,791-17,662m 
17,662-19,314m 
19,314-21,629m 
21,629-23,313m 
23,313-25,908m 



a 0.995a stands for the level at 0.995cr in sigma 
coordinate system. The sigma coordinate system 
is a vertical coordinate system being widely used 
in numerical models. It defines the vertical posi- 
tion ratio of the pressure difference between 
that position and the top of its belonging grid. 



Table 2. Coefficients for the AXP model as well as the corresponded GFS model layer and weight. 



AXP layer 



Corresponding GFS layer and weight 



p(h) 



A(h) 



(C 2 T )(h) 



0-50m above ground 
50- 100m above ground 
100-1, 000m above ground 
l-2km 



2- 3km 

3- 4km 

4- 5km 

5- 6km 

6- 7km 

7- 8km 

8- 9km 

9- 10km 

10- 11km 



2m above ground to 0.995u 

0.995(T to 30hPa above ground 
900- 



850 
800 
800- 
750- 
700 
650- 
650 
600 
550- 
550- 
500 
500- 
450- 
450- 
400 
400- 
350 
350 
300 
300 
250- 



-850hPa 
■800hPa 
•750hPa 
750hPa 
•700hPa 
■650hPa 
■600hPa 
600hPa 
■550hPa 
500hPa 
■500hPa 
450hPa 
450hPa 
■400hPa 
■400hPa 
■350hPa 
350hPa 
300hPa 
300hPa 
■250hPa 
■250hPa 
200hPa 



50%) 
40%) 
10%) 
50%) 
50%) 
60%) 
40%) 
20%) 
70%) 
10%) 
60%) 
40%) 
30%) 
70%) 
20%) 



10%) 
90%) 
20%) 
80%) 
40%) 
60%) 



P(/i)-3hPa 0.5 
P(/i)-9hPa 
P(/j)-60hPa 

845hPa 0.3 



1.6E+2 1.4E-2 

4.8E-4(/i/100)~ 2 - 6 
3.4E-5(/i/1000)- L1 



440hPa 



1.8E+3 



745hPa 1.3 5.6E+2 



655hPa 1.7 4.6E+2 



575hPa 1.7 3.8E+2 



505hPa 2.6 2.6E+2 



1.1 4.6E+2 



380hPa 0.8 6.8E+2 



330hPa 0.6 1.0E+3 



285hPa 0.3 2.2E+3 



245hPa 0.5 6.8E+2 



5.2E-5 

4.2E-5 
2.8E-5 
2.5E-5 

1.8E-5 
1.5E-5 
1.6E-5 
1.6E-5 
1.9E-5 
2.6E-5 



Table 2 — Continued 



AXP layer 


Corresponding GFS layer and weight 


P 


p(h) 


A(h) 




ll-12km 


250-200hPa (80%) 


210hPa 


0.7 


3.2E+2 


3.4E-5 




200-150hPa (20%) 










12-13km 


200-150hPa (100%) 


177hPa 


0.6 


2.2E+2 


4.4E-5 


13-14km 


200-150hPa (50%) 


150hPa 


0.1 


8.3E+3 


4.8E-5 




150-100hPa (50%) 










14-15km 


150-100hPa (100%) 


126hPa 


0.2 


1.0E+3 


5.5E-5 


15-16km 


150-100hPa (80%) 


105hPa 


-0.4 


3.2E+1 


6.5E-5 




100-70hPa (20%) 










16-17km 


100-70hPa (100%) 


88hPa 


-0.3 


1.5E+1 


8.3E-5 


17-18km 


100-70hPa (70%) 


72hPa 


2.5 


5.6E+1 


1.1E-4 




70-50hPa (30%) 










18-19km 


70-50hPa (100%) 


59hPa 


-0.9 


2.6E+1 


1.1E-4 


19-20km 


70-50hPa (30%) 


48hPa 


3.3 


3.8E+1 


9.5E-5 




50-30hPa (70%) 










9D 91 Irm 


OU-OUilJrd yL\J\J/0) 


oviir a 


1 n 




8 9F ^ 


21-22km 


50-30hPa (60%) 


31hPa 


1.5 


4.6E+1 


7.4E-5 




30-20hPa (40%) 










22-23km 


30-20hPa (100%) 


24hPa 


-1.9 


3.2E+1 


7.5E-5 


23-24km 


30-20hPa (30%) 


19hPa 


1.3 


4.6E+1 


8.7E-5 




20-10hPa (70%) 










24-25km 


20-10hPa (100%) 


15hPa 


1.1 


3.8E+1 


1.1E-4 


25-26km 


20-10hPa (100%) 


llhPa 


1.5 


3.8E+1 


1.3E-4 
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Table 3. Observing sites. PBL top stands for the upper limit of planetary boundary layer (PBL) adopted in the GFS/AXP 
model. P(h) is the atmospheric pressure at the height of the site used in GFS/AXP model, calculated under standard 

atmosphere. 



Site 


Location 


Elevation (h) 


GFS grid elevation 


Climate 


Data type 


Availability 


PBL top 


P(h) 


Par anal 


-70.40, -24.63 


2,635m 


919m 


Arid 


Cloud, DIMM 


2008 Jan. 1 - 2009 Dec. 31 


3,700m 


734hPa 


Mauna Kea 


-155.48, +19.83 


4,050m 


896m 


Highlands 


DIMM, MASS 


2008 Jan. 1- 2008 May 31 


5,100m 


612hPa 


San Pedro Martir 


-115.47, +31.05 


2,830m 


1,038m 


Arid 


DIMM, MASS 


2008 Jan. 1 - 2008 Aug. 31 


3,800m 


716hPa 


Cerro Tololo 


-70.80, -30.17 


2,200m 


926m 


Arid 


MASS 


2008 Jan. 1 - 2009 Dec. 31 


3,200m 


775hPa 


Nanshan 


+87.18, +43.47 


2,080m 


2,233m 


Semiarid 


Cloud 


2008 Jan. 1 - 2009 Jul. 4 






Lulin 


+120.87, +23.47 


2,862m 


1,345m 


Humid subtropical 


Cloud 


2008 Jan. 1 - 2009 Jul. 31 






Cerro Armazoncs 


-70.20, -24.60 


3,064m 


1,997m 


Arid 


DIMM, MASS 


2008 Jan. 1 - 2009 Sep. 30 


4,100m 


695hPa 


Cerro Pachon 


-70.73, -30.23 


2,715m 


2,706m 


Arid 


DIMM, MASS 


2008 Jan. 1 - 2009 Dec. 31 a 


3,700m 


727hPa 


Cerro Tolonchar 


-67.98, -23.93 


4,480m 


3,626m 


Highlands 


DIMM, MASS 


2008 Jan. 1 - 2008 Sep. 30 


5,500m 


579hPa 



Ground-based seeing observation from 2008 Oct. 31 to 2009 Apr. 29 only. 



-25- 



Table 4: Cloud cover forecast result for Paranal, Nanshan, and Lulin. 



Threshold 


Category 


Paranal 


Nanshan 


Lulin 


30% 


H 


73 


715 


682 




F 


443 


482 


860 




M 


71 


38 


14 




Z 


1411 


241 


94 




Total n 


1998 


1476 


1650 


50% 


H 


62 


684 


670 




F 


353 


386 


813 




M 


82 


69 


26 




Z 


1501 


337 


141 




Total n 


1998 


1476 


1650 


80% 


H 


44 


576 


642 




F 


225 


232 


734 




M 


100 


177 


54 




Z 


1629 


491 


220 




Total n 


1998 


1476 


1650 



Table 5: Evaluation result of cloud cover forecast for Paranal, Nanshan, and Lulin. 



Threshold Indicator Paranal (n=l,998) Nanshan (n=l,476) Lulin (n=l,650) 



30% 



PPF 
POD 
FAR 
FBI 



0.74 
0.51 



+0.03 
-0.02 
+0.12 



-0.07 

u ' Z4 -0.03 
o r:o+0.61 
°- oo -0.50 



fi^+ - 02 
U - DO -0.02 

h0.04 



0.05 



0.95] 

67+ - 01 
u -°' -o.oi 

1 59+ - 03 

i - oy -0.05 



0.47 
0.98 



+0.01 
-0.01 
+0.01 



-0.01 

90+ - 02 
u - yu -o.oi 

9 99 +0.01 
z - zz -0.03 



50% 



PPF 
POD 
FAR 
FBI 



n 7 o+o.03 
u -' 8-0.02 

n A-i+0.07 
U - 4,3 -0.04 
n iq+0.03 
u - iy -0.03 
9 OO+0.52 
z -°°-0.48 



0-6918;" 

u - oo -0.01 

h0.04 
-0.04 



1.32] 



0.491832 

0.9618;^ 

o.85i8;8^ 

9 iq+0.01 
z - lo -0.02 



80% 



PPF 
POD 
FAR 
FBI 



0-8418:81 
o.3H8:18 

12+ - 05 
u - iz -0.03 

i.86i8: 4 4 



72+ - 04 
u - ' z -0.03 



.+0.06 
-0.06 
hO.Ol 
-0.01 

1 07+ - 06 

i - u ' -0.06 



0.76] 
0.32^ 



^2 +0 - 02 

u - oz -0.02 

92+ - 05 

u - yz -0.02 
n 77 +0.02 
u - ' ' -0.02 

i - yo -0.05 
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Table 6: Mean difference (caicuiated by the mean of forecast minus the mean of observation) and 
RMSE of seeing forecast for the sample sites. 



Site 


Entire atmosphere 
Mean difference 


RMSE 


Sample n 


Free atmosphere 
Mean difference 


RMSE 


Sample n 


Par anal 


-0.09" 


0.36" 


3,630 








Mauna Kea 


-0.15" 


0.26" 


34 


-0.32" 


0.42" 


42 


San Pedro Martir 


+0.01" 


0.45" 


326 


-0.10" 


0.22" 


118 


Cerro Tololo 








-0.24" 


0.34" 


640 


Cerro Armazones 


+0.20" 


0.34" 


776 


-0.18" 


0.27" 


878 


Cerro Pachon 


+0.46" 


0.50" 


738 


-0.14" 


0.27" 


928 


Cerro Tolonchar 


+0.29" 


0.40" 


298 


-0.26" 


0.33" 


92 



Table 7: Comparison between the GFS/AXP model versus several other major seeing models. The 
values of the models of original AXP, Vernin-Tatarski, Cff median, seeing median, and seeing mean 
are taken from the work of Trinquet & Vernin. The probabilities of forecast with < 30% error for 
the cases of MKWC forecaster /WRF model are estimated manually from the plots generated on 
the MKWC website. 



Model 


Forecast lead time 


RMSE 


< 30% error probability 


GFS/AXP 


1-3 nights 


0.26"-0.50" 


Mostly 40-50% 


Original AXP 


Simultaneous 


0.48" 


~ 58% 


Vernin-Tatarski 


Simultaneous 




30% 


Cjy median 


Simultaneous 




41% 


Seeing median 


Simultaneous 




45% 


Seeing mean 


Simultaneous 




41% 


MKWC Forecaster 3, 


1-3 nights 


0.25"-0.30" 


~ 50%? 


MKWC WRF b 


1 night 


0.36" 


~ 50%? 



"See http:/ /mkwc.ifa.hawaii.edu/forecast/mko/stats/index.cgi?night=l 



http://mkwc.ifa. hawaii.edu/forecast/mko/stats/index.cgi?night=l 



Mcster=fcsts&var=seeing&cut=2. 
Mcster=wrf&var=seeing&cut=2. 



